Macro - Actions in Reinforcement Learning : An EmpiricalAnalysisAmy McGovern and Richard

Several researchers have proposed reinforcement learning methods that obtain advantages in learning by using temporally extended actions, or macro-actions, but none has carefully analyzed what these advantages are. In this paper, we separate and analyze two advantages of using macro-actions in reinforcement learning: the eeect on exploratory behavior, independent of learning, and the eeect on the speed with which the learning process propagates accurate value information. We empirically measure the separate contributions of these two eeects in gridworld and simulated robotic environments. In these environments, both eeects were signiicant, but the eeect of value propagation was larger. We also compare the accelerations of value propagation due to macro-actions and eligibility traces in the gridworld environment. Although eligibility traces increased the rate of convergence to the optimal value function compared to learning with macro-actions but without eligibility traces, eligibility traces did not permit the optimal policy to be learned as quickly as it was using macro-actions.